Decentralized learning works: An empirical comparison of gossip learning and federated learning

Machine learning over distributed data stored by many clients has important applications in use cases where privacy is a key concern or central storage not an option. Recently, federated was proposed to solve this problem. The assumption that the itself collected centrally. In master–worker architecture, workers perform machine their own and master merely aggregates resulting models without seeing any raw data, unlike parameter server approach. Gossip decentralized alternative does require aggregation indeed component. natural hypothesis gossip strictly less efficient than due relying on more basic infrastructure: only message passing no cloud resources. empirical study, we examine present systematic comparison of two approaches. experimental scenarios include real churn trace mobile phones, continuous bursty communication patterns, different network sizes distributions training devices. We also evaluate number additional techniques including compression technique based sampling, token account flow control for learning. aggregated cost both Surprisingly, best variants comparably overall, so they offer fully • Fully viable Compression essential all algorithms achieve competitive performance. Uneven class-label distribution nodes favors centralization. For communication, token-based improves convergence gossip.